👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Hi @yewentao256, thanks for reviewing my PR! As the gemini-code-assist point out, configuration file for E=64,N=768 may not be optimal as it was copied from the one for E=128,N=768. The benchmark script currently does not take expert-parallel into consideration therefore I wasn't able to generate a native configuration file for E=64,N=768. Is this indeed the case for current benchmark script and should I write custom script to find optimal config when loading model with --enable-expert-parallel?

@Chen-zexi I wasn't aware that EP hadn't been added to benchmark_moe.py.. would you be willing to add support for this?

@Chen-zexi I wasn't aware that EP hadn't been added to benchmark_moe.py.. would you be willing to add support for this?

Sure, I'd happy to.

In the meantime, can you show some e2e benchmarks that these configs actually make the deployment faster in your hardware?

Hi @mgoin, I am getting the following results running the command below:

python3 benchmarks/benchmark_throughput.py \
    --model Qwen/Qwen3-30B-A3B \
    --tensor-parallel-size 2 \
    --dataset-name sonnet \
    --dataset-path benchmarks/sonnet.txt \
    --max-num-batched-tokens 4096
    --num-prompts 100

Without tuned config:

INFO 07-02 11:37:18 [metrics.py:417] Avg prompt throughput: 10147.5 tokens/s, Avg generation throughput: 1061.8 tokens/s, Running: 100 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.0%, CPU KV cache usage: 0.0%.

INFO 07-02 11:37:22 [multiproc_worker_utils.py:260] Worker exiting
Throughput: 11.16 requests/s, 7729.43 total tokens/s, 1674.10 output tokens/s
Total num prompt tokens:  54256
Total num output tokens:  15000

With tuned config:

INFO 07-02 11:43:38 [metrics.py:417] Avg prompt throughput: 10822.0 tokens/s, Avg generation throughput: 1371.7 tokens/s, Running: 100 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.1%, CPU KV cache usage: 0.0%.

INFO 07-02 11:43:42 [multiproc_worker_utils.py:260] Worker exiting
Throughput: 11.13 requests/s, 7708.03 total tokens/s, 1669.46 output tokens/s
Total num prompt tokens:  54256
Total num output tokens:  15000

I assume it means higher throughput when using the tuned config during runtime, however overall throughputs are pretty identical.

Might worth mention, I am seeing lower throughputs when EP is enabled:

INFO 07-02 11:49:38 [metrics.py:417] Avg prompt throughput: 10800.5 tokens/s, Avg generation throughput: 970.8 tokens/s, Running: 100 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.9%, CPU KV cache usage: 0.0%.

INFO 07-02 11:49:42 [multiproc_worker_utils.py:260] Worker exiting
Throughput: 10.55 requests/s, 7303.80 total tokens/s, 1581.91 output tokens/s
Total num prompt tokens:  54256
Total num output tokens:  15000

Does these number look right to you?

Hi @mgoin , I made a PR #20501 with EP support to benchmark_moe.py
Below is some result I got:

Benchmark result generated by:

python3 benchmarks/benchmark_serving.py --model Qwen/Qwen3-30B-A3B \
    --dataset-name sonnet \
    --dataset-path benchmarks/sonnet.txt \
    --num-prompts 1000 \
    --save-result

Hi @yewentao256, thank you for reviewing my PR. I tested with two Pro 6000 Blackwell Workstation GPUs on a consumer-grade motherboard, which only supports PCIe 5.0 x8 per GPU in a dual-GPU setup. As a result, the bottleneck is most likely due to limited bandwidth for GPU-to-GPU communication—this is especially the case for EP. I expect performance would improve significantly with GPUs that support NVLink, but unfortunately, I do not have access to additional GPUs to test this further.

The configuration files I uploaded should be fine for now, as this particular GPU does not support NVLink anyway.